Systems and methods for generating composite sets of data from different sensors

ABSTRACT

Systems and methods for generating composite sets of data based on sensor data from different sensors are disclosed. Exemplary implementations may capture a color image including chromatic information; capture a depth image; generate inertial signals conveying values that are used to determine motion parameters; determine the motion parameters based on the inertial signals; generate a re-projected depth image as if the depth image had been captured at the same time as the color image, based on the interpolation of motion parameters; and generate a composite set of data based on different kinds of sensor data by combining information from the color image, the re-projected depth image, and one or more motion parameters.

FIELD OF THE DISCLOSURE

The present disclosure relates to systems and methods for generatingcomposite sets of data from different sensors, e.g., different types ofsensors having different rates of operation.

BACKGROUND

Image sensors, depth sensors, inertial sensors, thermal sensors, andother sensors are known. Depth images as captured by a depth sensor areknown. Determining the movement an object has made based on signals froman inertial sensor (coupled to the object) is known.

SUMMARY

One aspect of the present disclosure relates to a system configured forgenerating composite sets of data based on sensor data from differentsensors. The system may include one or more hardware processorsconfigured by machine-readable instructions. The system may beconfigured to capture, by an image sensor, images from viewpoints, theimages including chromatic information. The chromatic information ofindividual images may indicate one or more colors viewable by the imagesensor from individual viewpoints of the image sensor. The images mayinclude a first image captured at a first image capture time from afirst image viewpoint. The system may be configured to capture, by adepth sensor, depth images from viewpoints of the depth sensor. Thedepth images may include depth information. The depth information ofindividual depth images may be captured from individual viewpoints ofthe depth sensor. The depth information of the individual depth imagesmay indicate distances from the individual viewpoints to surfacesviewable by the depth sensor from the individual viewpoints. The depthimages may include a first depth image including first depthinformation. The first depth information may be captured from a firstdepth viewpoint at a first depth-capture time. The first depthinformation may indicate a first set of distances from the first depthviewpoint to the surfaces. The system may be configured to generate, byan inertial sensor, inertial signals that convey values that are used todetermine motion parameters characterizing position and orientation ofthe inertial sensor in a reference coordinate system. The inertialsignals may include a first set of inertial signals generated at a firstinertial-sensor-measurement time that convey a first set of values thatis used to determine a first set of motion parameters and a second setof inertial signals generated at a second inertial-sensor-measurementtime that convey a second set of values that is used to determine asecond set of motion parameters. The processor(s) may be configured todetermine the first set of values of the first set of one or more motionparameters based on the first set of inertial signals. The second set ofvalues of the second set of one or more motion parameters may be based(at least in part) on the second set of inertial signals. A set ofvalues of one or more interpolated motion parameters (also referred toas an interpolated set) may be based on the first set of values and thesecond set of values. The interpolated set of values may correspond to apoint in time between the first inertial-sensor-measurement time and thesecond inertial-sensor-measurement time (in particular, at the firstimage capture time). The processor(s) may be configured to generate afirst re-projected depth image representing the first depth informationincluded in the first depth image as if the first depth image had beencaptured at a point in time between the firstinertial-sensor-measurement time and the secondinertial-sensor-measurement time (in particular, at the first imagecapture time). Generation of the first re-projected depth image may bebased on the interpolated set of values. The processor(s) may beconfigured to generate a composite set of data by combining informationfrom the first image, the first re-projected depth image, and theinterpolated set of one or more interpolated motion parameters.

Another aspect of the present disclosure relates to a method forgenerating composite sets of data based on sensor data from differentsensors. The method may include capturing, by an image sensor, imagesfrom viewpoints, the images including chromatic information. Thechromatic information of individual images may indicate one or morecolors viewable by the image sensor from individual viewpoints of theimage sensor. The images may include a first image captured at a firstimage capture time from a first image viewpoint. The method may includecapturing, by a depth sensor, depth images from viewpoints of the depthsensor. The depth images may include depth information. The depthinformation of individual depth images may be captured from individualviewpoints of the depth sensor. The depth information of the individualdepth images may indicate distances from the individual viewpoints tosurfaces viewable by the depth sensor from the individual viewpoints.The depth images may include a first depth image including first depthinformation. The first depth information may be captured from a firstdepth viewpoint at a first depth-capture time. The first depthinformation may indicate a first set of distances from the first depthviewpoint to the surfaces. The method may include generating, by aninertial sensor, inertial signals that convey values that are used todetermine motion parameters characterizing position and orientation ofthe inertial sensor in a reference coordinate system. The inertialsignals may include a first set of inertial signals generated at a firstinertial-sensor-measurement time that convey a first set of values thatis used to determine a first set of motion parameters and a second setof inertial signals generated at a second inertial-sensor-measurementtime that convey a second set of values that is used to determine asecond set of motion parameters. The method may include determining thefirst set of values of the first set of one or more motion parametersbased on the first set of inertial signals. The second set of values ofthe second set of one or more motion parameters may be based (at leastin part) on the second set of inertial signals. An interpolated set ofvalues of one or more interpolated motion parameters may be based on thefirst set of values and the second set of values. The interpolated setof values may correspond to a point in time between the firstinertial-sensor-measurement time and the secondinertial-sensor-measurement time (in particular, at the first imagecapture time). The method may include generating a first re-projecteddepth image representing the first depth information included in thefirst depth image as if the first depth image had been captured at apoint in time between the first inertial-sensor-measurement time and thesecond inertial-sensor-measurement time (in particular, at the firstimage capture time). Generation of the first re-projected depth imagemay be based on the interpolated set of values. The method may includegenerating a composite set of data by combining information from thefirst image, the first re-projected depth image, and the interpolatedset of one or more interpolated motion parameters.

As used herein, any association (or relation, or reflection, orindication, or correspondency) involving servers, processors, clientcomputing platforms, sensors, images, viewpoints, viewing angles,capture times, signals, values, parameters, positions, orientations,and/or another entity or object that interacts with any part of thesystem and/or plays a part in the operation of the system, may be aone-to-one association, a one-to-many association, a many-to-oneassociation, and/or a many-to-many association or N-to-M association(note that N and M may be different numbers greater than 1).

As used herein, the term “obtain” (and derivatives thereof) may includeactive and/or passive retrieval, determination, derivation, transfer,upload, download, submission, and/or exchange of information, and/or anycombination thereof. As used herein, the term “effectuate” (andderivatives thereof) may include active and/or passive causation of anyeffect. As used herein, the term “determine” (and derivatives thereof)may include measure, calculate, compute, estimate, approximate,generate, and/or otherwise derive, and/or any combination thereof. Asused herein, the term “composite” refers to a combination of differentkinds of information, including but not limited to captured information,generated information, and interpolated information.

These and other features, and characteristics of the present technology,as well as the methods of operation and functions of the relatedelements of structure and the combination of parts and economies ofmanufacture, will become more apparent upon consideration of thefollowing description and the appended claims with reference to theaccompanying drawings, all of which form a part of this specification,wherein like reference numerals designate corresponding parts in thevarious figures. It is to be expressly understood, however, that thedrawings are for the purpose of illustration and description only andare not intended as a definition of the limits of the invention. As usedin the specification and in the claims, the singular form of “a”, “an”,and “the” include plural referents unless the context clearly dictatesotherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system configured for generating composite sets ofdata from different sensors, in accordance with one or moreimplementations.

FIG. 2 illustrates a method for generating composite sets of data fromdifferent sensors, in accordance with one or more implementations.

FIG. 3A-3B-3C-3D illustrate exemplary scenarios for the use of a systemconfigured for generating composite sets of data from different sensors,in accordance with one or more implementations.

FIG. 4A-4B-4C illustrate exemplary depth images from elevated viewpointsas may be used by a system configured for generating composite sets ofdata from different sensors, in accordance with one or moreimplementations.

FIG. 5 illustrates a spherical coordinate system as may be used by asystem configured for generating composite sets of data from differentsensors, in accordance with one or more implementations.

FIG. 6A-6B illustrate exemplary scenarios for the use of a systemconfigured for generating composite sets of data from different sensors,in accordance with one or more implementations.

FIG. 7A-7B-7C illustrate exemplary timelines with sensor operations forthe use of a system configured for generating composite sets of datafrom different sensors, in accordance with one or more implementations.

FIG. 8 illustrates an exemplary timeline with composite sets of data, inaccordance with one or more implementations.

FIG. 9A-9B-9C-9D illustrate exemplary timelines with sensor operationsfor the use of a system configured for generating composite sets of datafrom different sensors, in accordance with one or more implementations.

DETAILED DESCRIPTION

FIG. 1 illustrates a system 100 configured for generating composite setsof data based on sensor data from different sensors, in accordance withone or more implementations. Different sensors may be rigidly attachedto each other or to a structural component of system 100 (such as, byway of non-limiting example, a rig) such that their relative positionsare fixed an unchanging even if one or more of the sensors move. In someimplementations, system 100 may include one or more servers 102, animage sensor 108, a depth sensor 110, an inertial sensor 112, a thermalsensor 120, electronic storage 128, processor(s) 130, and/or othercomponents. Server(s) 102 may be configured to communicate with one ormore client computing platforms 104 according to a client/serverarchitecture and/or other architectures. Client computing platform(s)104 may be configured to communicate with other client computingplatforms via server(s) 102 and/or according to a peer-to-peerarchitecture and/or other architectures. Users may access system 100 viaclient computing platform(s) 104. In some implementations, system 100may include an augmented reality device 132.

Different sensors may have different rates of operation. For example, afirst sensor may generate signals at a first rate, a second sensor maygenerate signals at a second rate, a third sensor may generate signalsat a third rate, and so forth. The sensors may be (rigidly and/orjointly) moving during operation, the movements including one or more oftranslation, rotation, and/or other movements. The signals generated bydifferent sensors (and the information and/or parameters conveyed bythese signals) may not be aligned temporally. For example, even if twosensors both generate signals at a frequency of 60 Hz, these signals maybe temporally misaligned. In other words, these equal-rate signals maybe generated at different points in time. Different points in time maycorrespond to different points in space. Even if multiple sensors usethe same clock signal for timestamping generated data, the signalsgenerated by the multiple sensors may still be generated at differentpoints in time and/or space. By virtue of the technologies disclosedherein, sensor data from different sensors may be combined in compositesets of data that are temporally aligned as if they were produced bytriggering the different sensors at the same time.

Image sensor 108 may be configured to capture images. Individual imagesmay be captured from individual viewpoints. As used herein, a viewpointmay be defined by a combination of a three-dimensional position and athree-dimensional orientation in a reference coordinate system. In someimplementations, individual images may be captured from individualorientations. In some implementations, individual images may be capturedfrom individual positions. In some implementations, image sensor 108 maybe configured to capture individual images at a particular capture rate,e.g., an image-capture rate of 20 Hz, 30 Hz, 40 Hz, 50 Hz, 60 Hz, 70 Hz,80 Hz, 90 Hz, 100 Hz, and/or another capture rate. In someimplementations, the position of a viewpoint may be represented by alocation in three-dimensional space, such as, e.g., a referencecoordinate system.

In some implementations, a viewpoint may be represented by a point in athree-dimensional reference coordinate system that characterizes theposition of the image sensor 108 in the reference coordinate system, aswell as by a three-dimensional vector in the reference coordinate systemthat characterizes the orientation of the image sensor 108 in thereference coordinate system. Likewise, a two-dimensional coordinatesystem may be used for reference. In some implementations, the locationof image sensor 108 may correspond to the optical center of the imagesensor 108, such as the lens center or the aperture center. In someimplementations, this location may correspond to the origin of the imagesensor (local) coordinate system. The direction of the orientationvector may be referred to as the viewing direction or the optical axisof image sensor 108. In some implementations, the orientation vector maycoincide with one of the axis of the image-sensor (local) coordinatesystem. For drawing purposes, the starting point of the orientationvector may correspond to the viewpoint position of image sensor 108.

By way of non-limiting example, FIG. 3A illustrates an exemplaryscenario 30 for the use of system 100, depicting a top view of imagesensor 108 having a viewpoint 108 a. The boundary of observable spacefor image sensor 108 in scenario 30 is depicted by boundaries 30 a and30 b. As used herein, the term observable may refer to space that isdepicted in a captured image. In some implementations, viewing direction30 c may correspond to the center of observable space for image sensor108, which may be centered between boundaries 30 a and 30 b. The anglebetween boundaries 30 a and 30 b may represent a horizontalangle-of-view 108 h.

By way of non-limiting example, FIG. 3B illustrates an exemplaryscenario 31 for the use of system 100, depicting a side view of imagesensor 108 having a viewpoint 108 a. The boundary of observable spacefor image sensor 108 in scenario 31 is depicted by boundaries 31 a and31 b. Viewing direction 31 c may correspond to the center of observablespace for image sensor 108, which may be centered between boundaries 31a and 31 b. The angle between boundaries 31 a and 31 b may represent avertical angle-of-view 108 v. In some implementations, viewing direction30 c in FIG. 3A may coincide which viewing direction 31 c in FIG. 3B.

By way of non-limiting example, FIG. 3C illustrates an exemplaryscenario 32 for the use of system 100, depicting an isometric and/orthree-dimensional view of image sensor 108 having a viewpoint 108 a. Theboundaries of observable space for image sensor 108 in scenario 32 isdepicted by boundaries 32 a and 32 b, which may form a shape similar toa cone. Viewing direction 32 c may correspond to the center ofobservable space for image sensor 108, which may be centered betweenboundaries 32 a and 32 b.

By way of non-limiting example, FIG. 3D illustrates an exemplaryscenario 33 for the use of system 100, depicting an isometric and/orthree-dimensional view of image sensor 108 having a viewpoint 108 a. Theboundary of observable space for image sensor 108 in scenario 33 isdepicted by boundaries 33 a, 33 b, 33 c, and 33 d, which may form ashape similar to a pyramid. For example, the corners of boundaries 33 a,33 b, 33 c, and 33 d may correspond to the corners (or corner pixels) ofthe image based on the information captured by image sensor 108. Fromthe perspective of a viewer of a rectangular image within thisobservable space, boundary 33 a may be the top right corner of theimage, boundary 33 b may be the top left corner of the image, boundary33 c may be the bottom right corner of the image, and boundary 33 d maybe the bottom left corner of the image. Viewing direction 33 ecorresponds to the center of observable space for image sensor 108 inscenario 33, which may be centered between boundaries 33 a, 33 b, 33 c,and 33 d.

Referring to FIG. 1 and image sensor 108, individual images may includechromatic information. The chromatic information of individual imagesmay indicate one or more colors viewable by the image sensor fromindividual viewpoints of the image sensor. In some implementations, thechromatic information may be a black and white image. In someimplementations, the chromatic information may be a color image, e.g.,an image including red, green, and blue information. Other color formatsare envisioned within the scope of this disclosure. The images mayinclude a first image captured at a first image capture time from afirst image viewpoint, a second image captured at a second image capturetime from a second image viewpoint, a third image captured at a thirdimage capture time from a third image viewpoint, and so forth.

Image sensor 108 may include, by way of non-limiting example, one ormore of an image sensor, a camera, and/or another sensor. In someimplementations, image sensor 108 may be physically and rigidly coupledto one or more other sensors, and/or another component of system 100.Accordingly, information from inertial sensor 112 may not only reflectmotion of inertial sensor 112, but may also correspond in a known mannerto motion of image sensor 108 and/or other components of system 100, dueto their known relative position and orientation. In someimplementations, one or more of the sensors of system 100 may include analtimeter (e.g., a sonic altimeter, a radar altimeter, and/or othertypes of altimeters), a barometer, a magnetometer, a pressure sensor(e.g., a static pressure sensor, a dynamic pressure sensor, a pitotsensor, etc.), a thermometer, an accelerometer, a gyroscope, an inertialmeasurement sensor, a geolocation sensor, global positioning systemsensors, a tilt sensor, a motion sensor, a vibration sensor, adistancing sensor, an ultrasonic sensor, an infrared sensor, a lightsensor, a microphone, an air speed sensor, a ground speed sensor, analtitude sensor, degree-of-freedom sensors (e.g., 6-DOF and/or 9-DOFsensors), a compass, and/or other sensors. As used herein, the term“motion sensor” may include one or more sensors configured to generateoutput conveying information related to position, location, distance,motion, movement, acceleration, jerk, jounce, and/or other motion-basedparameters.

Output signals generated by individual sensors (and/or information basedthereon) may be stored and/or transferred in electronic files. In someimplementations, output signals generated by individual sensors (and/orinformation based thereon) may be streamed to one or more othercomponents of system 100.

As used herein, the terms “camera” and/or “image sensor” may include anydevice that captures images, including but not limited to a singlelens-based camera, a wide-lens camera, a camera array, a solid-statecamera, a mechanical camera, a digital camera, an image sensor, a depthsensor, a remote sensor, a lidar, an infrared sensor, a (monochrome)complementary metal-oxide-semiconductor (CMOS) sensor, an active pixelsensor, and/or other sensors. Image sensor 108 may be configured tocapture information, including but not limited to visual information,video information, audio information, geolocation information,orientation and/or motion information, depth information, and/or otherinformation. Information captured by sensors may be marked, timestamped,annotated, and/or otherwise processed such that information captured byother sensors can be synchronized, aligned, annotated, and/or otherwiseassociated therewith. For example, video information captured by animage sensor may be synchronized with information captured by anaccelerometer, GPS unit, and/or one or more other sensors.

In some implementations, an image sensor may be integrated withelectronic storage such that captured information may be stored, atleast initially, in integrated embedded storage. For example, a cameramay include one or more image sensors and electronic storage media. Insome implementations, an image sensor may be configured to transfercaptured information to one or more components of system 100, includingbut not limited to remote electronic storage media, e.g., through “thecloud.”

Depth sensor 110 may be configured to capture depth images. Individualimages may be captured from individual viewpoints of depth sensor 110.In some implementations, individual depth images may be captured fromindividual orientations of depth sensor 110 (also referred to asindividual depth orientations). In some implementations, individualdepth images may be captured from individual positions of depth sensor110 (also referred to as individual depth positions). In someimplementations, depth sensor 110 may be configured to captureindividual depth images at a particular capture rate, e.g., adepth-capture rate of 10 Hz, 20 Hz, 30 Hz, 40 Hz, 50 Hz, 60 Hz, 70 Hz,80 Hz, 90 Hz, 100 Hz, and/or another depth-capture rate. In someimplementations, depth sensor 110 may be one or more of astructured-light active stereo sensor, a passive stereo sensor, acontinuous-wave time-of-flight (TOF) range sensor, a pulsed-light TOFsensor, and/or one or more other types of depth sensors.

The depth images may include depth information. The depth information ofindividual depth images may be captured from individual viewpoints ofdepth sensor 110. The depth images may include a first depth imageincluding first depth information, a second depth image including seconddepth information, a third depth image including third depthinformation, and so forth. The first depth information may be capturedfrom a first depth viewpoint at a first depth-capture time. The seconddepth information may be captured from a second depth viewpoint at asecond depth-capture time. The third depth information may be capturedfrom a third depth viewpoint at a third depth-capture time, and soforth.

In some implementations, the depth information of the individual depthimages may indicate distances from the individual viewpoints to surfacesviewable by the depth sensor from the individual viewpoints. Forexample, the first depth information may indicate a first set ofdistances from the first depth viewpoint to the surfaces. In someimplementations, the depth information of an individual element of adepth image may be a three-dimensional position, and the depthinformation of the entire depth image may form a three-dimensional pointcloud. By way of non-limiting example, FIG. 5 illustrates a sphericalcoordinate system 50 for three-dimensional space (having an x-axis,y-axis, and z-axis), where the position of any point is specified bythree numbers: the radial distance of that point from a fixed origin,its polar angle measured from a fixed zenith direction, and the azimuthangle of its orthogonal projection on a reference plane that passesthrough the origin and is orthogonal to the zenith, measured from afixed reference direction on that plane. As depicted in sphericalcoordinate system 50, these three numbers may be referred to by the setof symbols (r, θ, φ), which gives the radial distance, polar angle, andazimuthal angle. To define a spherical coordinate system, one mustchoose two orthogonal directions, the zenith and the azimuth reference,and an origin point in space. These choices determine a reference planethat contains the origin and is perpendicular to the zenith. Thespherical coordinates of a point P are then defined as follows:

-   -   The radius or radial distance is the Euclidean distance from the        origin O to P.    -   The inclination (or polar angle) is the angle between the zenith        direction and the line segment OP.    -   The azimuth (or azimuthal angle) is the signed angle measured        from the azimuth reference direction to the orthogonal        projection of the line segment OP on the reference plane.

The sign of the azimuth may be determined by choosing what is a positivesense of turning about the zenith. This choice is arbitrary, and is partof the coordinate system's definition.

In some implementations, the depth information of an individual depthimage may indicate distances (e.g., radial distances) from a particulardepth viewpoint of depth sensor 110. A particular radial distance of anindividual element of a depth image may correspond to a particular polarangle and a particular azimuth angle. Other coordinate systems areenvisioned within the scope of this disclosure, including but notlimited to non-spherical coordinate systems such as, for example,Cartesian Coordinates (Euclidean space), and/or other coordinatesystems. By way of non-limiting example, FIG. 4A illustrates anexemplary depth image 40 from an elevated viewpoint. Exemplary depthimage 40 indicates distances from a particular depth viewpoint 45. Theboundaries of observable space of depth sensor 110 from particular depthviewpoint 45 is depicted by boundaries 40 a, 40 b, 40 c, and 40 d, whichform a shape similar to a pyramid. From the perspective of a viewer ofexemplary depth image 40, boundary 40 c may be the top right corner ofthe image, boundary 40 d may be the top left corner of the image,boundary 40 a may be the bottom right corner of the image, and boundary40 b may be the bottom left corner of exemplary depth image 40. In someimplementations, the elements in a depth image may be arranged in agrid, in a similar arrangement as pixels in color images. For example,as depicted in FIG. 4A, the top row of elements in exemplary depth image40 may include elements 40 f, 40 g, 40 h, 40 i, 40 j, 40 k, and 40L. Thenumber of elements depicted is exemplary and not intended to be limitingin any way. The resolution of depth images is not limited by any figurein this disclosure. An individual element of a depth image maycorrespond to a particular polar angle and a particular azimuth angle.Additionally, individual elements may be associated with a depth value(e.g., a radial distance to particular depth viewpoint 45). If exemplarydepth image 40 is a flat surface arranged orthogonally to a viewingdirection of depth sensor 110, elements closer to the corners ofobservable space (such as elements 40 f and 40L) will have a greaterradial distance than elements further from the corners (such as elements40 h, 40 i, and 40 j). By way of non-limiting example, FIG. 4Billustrates an exemplary depth image 41 from an elevated viewpoint,similar to exemplary depth image 40 in FIG. 4A. From the perspective ofa viewer of exemplary depth image 41, boundary 40 a may be the bottomright corner of the image, and boundary 40 b may be the bottom leftcorner of exemplary depth image 40. In some implementations, theelements in a depth image may be arranged in a grid. For example, asdepicted in FIG. 4B, the center row of elements in exemplary depth image40 may be set of elements 40 m, such that individual elements in set 40m have different viewing angles. The number of elements depicted isexemplary and not intended to be limiting in any way. By way ofnon-limiting example, FIG. 4C illustrates an exemplary depth image 42from an elevated viewpoint, similar to exemplary depth image 40 in FIG.4A. From the perspective of a viewer of exemplary depth image 42,boundary 40 a may be the bottom right corner of the image, and boundary40 b may be the bottom left corner of exemplary depth image 40. In someimplementations, the elements in a depth image may be arranged in agrid. For example, as depicted in FIG. 4C, the bottom row of elements inexemplary depth image 42 may be set of elements 40 n, such thatindividual elements in set 40 n have different viewing angles. Thenumber of elements depicted is exemplary and not intended to be limitingin any way. The exemplary depth images in FIGS. 4A-4B-4C may be combinedsuch that the superset of depicted elements forms a grid.

Depth sensor 108 may be moving while capturing depth images. As depthsensor 110 moves, it may also rotate, e.g. in three dimensions. By wayof non-limiting example, depth sensor 110 may be a consumer-grade depthsensor, such as the INTEL™ REALSENSE™ R200. In some implementations,inertial sensor 112 may be physically and rigidly coupled to depthsensor 110, image sensor 108, and/or another component of system 100.Accordingly, information from inertial sensor 112 may not only reflectmotion of inertial sensor 112, but may also correspond in a known mannerto motion of depth sensor 110, image sensor 108, and/or anothercomponent of system 100.

Referring to FIG. 1, inertial sensor 112 may be configured to captureand/or generate inertial signals. The inertial signals may convey valuesthat are used to determine motion parameters. The motion parameters maycharacterize position, orientation, and/or other characteristics ofinertial sensor 112, including characteristics pertaining to movementand/or position in a reference coordinate system. In someimplementations, one or more motion parameters may include derivativesof position, orientation, and/or other characteristics of inertialsensor 112. In some implementations, one or more motion parameters maybe referred to as “absolute,” i.e., with respect to a referencecoordinate system. In some implementations, one or more motionparameters may be referred to as “relative,” i.e., with respect to othermotion parameters (e.g., a first orientation and a second orientationmay be used as relative to each other). The inertial signals may includea first set of inertial signals generated at a firstinertial-sensor-measurement time, a second set of inertial signalsgenerated at a second inertial-sensor-measurement time, a third set ofinertial signals generated at a third inertial-sensor-measurement time,and so forth. In some implementations, the first set of inertial signalsmay convey a first set of values that is used to determine a first setof (absolute) motion parameters, the second set of inertial signals mayconvey a second set of values that is used to determine a second set of(absolute) motion parameters, the third set of inertial signals mayconvey a third set of values that is used to determine a third set of(absolute) motion parameters, and so forth. In some implementations, twoor more absolute motion parameters (of the inertial sensor) may be usedto determine the relative motion parameters between twoinertial-sensor-measurement times. In some implementations, theseabsolute motion parameters may be used to determine the absolute motionparameters (and/or the relative motion parameters) of any other sensor.In some implementations, the first set of motion parameters may includethe same parameters as the second set on motion parameters, and/or thethird set of motion parameters (such as, e.g., an absolute positionand/or an absolute orientation). In some implementations, the firstinertial-sensor-measurement time may occur before a particular imagecapture time. In some implementations, the secondinertial-sensor-measurement time may occur after the particular imagecapture time. The first set of values of the first set of motionparameters may include a first position and a first orientation in thereference coordinate system. The second set of values of the second setof motion parameters may include a second position and a secondorientation in the reference coordinate system. The third set of valuesof the third set of motion parameters may include a third position and athird orientation in the reference coordinate system.

In some implementations, inertial sensor 112 may be or include aninertial measurement unit (IMU). In some implementations, inertialsensor 112 may include a gyroscope. In some implementations, theparameters may include angular velocity and/or a parameter based on orrelated to angular velocity. Alternatively, and/or simultaneously, insome implementations, inertial sensor 112 may include an accelerometer.In some implementations, the parameters may include acceleration and/ora parameter based on or related to acceleration. As used herein,acceleration may include two-dimensional acceleration, three-dimensionalacceleration, angular acceleration, and/or other types of acceleration.For example, in some implementations, the parameters may include one ormore of yaw rate, roll rate, and/or pitch rate. In some implementations,inertial sensor 112 may be configured to process inertial informationand/or signals and provide, at a particular rate, an absoluteorientation, absolute position, and/or other absolute motion parameterswithin a reference coordinate system. For example, the particular ratemay be 30 Hz, 60 Hz, 90 Hz, 120 Hz, 150 Hz, 180 Hz, and/or another rate.In some implementations, inertial sensor 112, IMU, and/or anothercomponent of system 100 may be configured to provide derivatives ofrotation and/or translation such that absolute motion parameters may bedetermined by integrating one or more derivatives. In someimplementations, an external system may remove bias from the generatedoutput signals by inertial sensor 112. In some implementations, such anexternal system may use a Kalman filter and/or other filters to filterand/or otherwise preprocess the generated output signals, and, e.g.,provide absolute motion parameters.

Thermal sensor 120 may be configured to capture thermal images includingthermal information. The thermal images may include a first thermalimage captured from a particular viewpoint and a particular capturetime. In some implementations, elements of the thermal information maybe arranged in a grid.

Server(s) 102 may be configured by machine-readable instructions 106.Machine-readable instructions 106 may include one or more instructioncomponents. The instruction components may include computer programcomponents. The instruction components may include one or more ofreprojection component 116, alignment component 118, image capturecomponent 120, thermal projection component 122, display component 124,and/or other instruction components.

Parameter determination component 114 may be configured to determinesets of values of motion parameters. Determinations by parameterdetermination component 114 may be based on signals generated and/orprovided by other sensors, such as inertial sensor 112. For example, afirst set of values of motion parameters may be based on a first set ofinertial signals, a second set of values of motion parameters may bebased on a second set of inertial signals, and so forth. In someimplementations, parameter determination component 114 may be configuredto determine an interpolated set of values of (absolute or relative)interpolated motion parameters based on multiple sets of values ofmotion parameters, such as, for example, a first set of values and asecond set of values. For example, the sets of values may include afirst set of values for a first position and a first orientation ofinertial sensor 112 (and/or another component of system 100) and asecond set of values for a second position and a second orientation ofinertial sensor 112 (and/or another component of system 100). In someimplementations, the interpolated set of values may include aninterpolated position (e.g., based on interpolating the first positionand the second position) and an interpolated orientation (e.g., based oninterpolating the first orientation and the second orientation) in areference coordinate system. The interpolated set of values maycorrespond to a point in time between the firstinertial-sensor-measurement time and the secondinertial-sensor-measurement time. In some implementations, the point intime of the interpolated set of values may coincide with a particularimage capture time. In some implementations, the interpolated positionmay coincide with a particular image viewpoint. By way of non-limitingexample, FIG. 6A-6B illustrate exemplary scenarios includinginterpolation, as described herein. FIG. 6A illustrates a scenario 60,depicted from a top view, in which inertial sensor 112 moves in timenear a stationary object 61. As depicted, the position of inertialsensor 112 changes over time, as the time progresses from a timeindicated by t=0 to a subsequent time indicated by t=1, to a subsequenttime indicated by t=2. During this movement, inertial sensor 112rotates, such that an orientation 112 a at time t=0 progresses to anorientation 112 b at time t=1 and subsequently to an orientation 112 cat time t=2. FIG. 6B illustrates a similar scenario, scenario 62, inwhich a position of a moving sensor changes over time, starting at aposition 112 d (corresponding to a time t=0 and orientation 112 a),progressing to a position 112 e (corresponding to a time t=1 andorientation 112 b), and subsequently to a position 112 f (correspondingto a time t=2 and orientation 112 c). Position 112 d and position 112 emay be interpolated to create and/or determine interpolated position 113(corresponding to an interpolated orientation 113 a), e.g., betweenpositions 112 d and 112 e. For example, such a position may correspondto any point in time between t=0 and t=1, e.g. t=0.5. Likewise, position112 e and position 112 f may be interpolated to create and/or determineinterpolated position 114 (corresponding to an interpolated orientation114 a), e.g., between positions 112 e and 112 f. Orientation 112 a andorientation 112 b may be interpolated to create and/or determineorientation 113 a that corresponds to interpolated position 113, andorientation 112 b and orientation 112 c may be interpolated to createand/or determine an orientation 114 a that corresponds to interpolatedposition 114. In some implementations, more than two positions and/ororientations may be used to interpolate at a specific point in time.

By way of non-limiting example, FIG. 7A-7B illustrate exemplarytimelines with sensor operations that are interpolated, as describedherein. FIG. 7A illustrates a timeline 70 depicting the occurrences ofdifferent sensor operations at different times. First, measurement 70 aoccurs, when inertial sensor 112 generates a first set of signalsconveying values that is used to determine a first position and a firstorientation in a reference coordinate system. Subsequently, measurement70 b occurs, when depth sensor 110 captures a first depth image.Subsequently, measurement 70 c occurs, when image sensor 108 captures afirst (color) image. Finally, measurement 70 d occurs, when inertialsensor 112 generates a second set of signals conveying values that isused to determine a second position and a second orientation in thereference coordinate system. FIG. 7B illustrates a timeline 71 that issimilar to timeline 70 in FIG. 7A, with the addition of interpolation 71a, which represents an interpolation of motion parameters based onmeasurements 70 a and 70 d to create an interpolated position and aninterpolated orientation (a set 71 x of interpolated parameters). Thisinterpolation may be weighted to coincide with the point in time of theoccurrence of measurement 70 c. In some implementations, a set of motionparameters can be interpolated to create an interpolated positioncorresponding to a time that coincides with another measurement orcapture (e.g., with the capture of an image such as a color image)provided that at least one motion parameter is determined at a momentthat occurred prior to the capture time of the image and at least onemotion parameter is determined at a moment that occurred subsequent tothe capture time of the image. For example, in some implementations,three or four or more than four motion parameters may be used for theinterpolation.

Referring to FIG. 1, in some implementations, parameter determinationcomponent 114 may be configured to determine the relative motionparameters of the depth sensor 110 from any depth timestamp to any pointin time that interpolation occurs. For example, the relative motionparameters may be the rotational and positional change of the depthsensor from the first depth timestamp to a subsequent point in time ofinterpolation occurrence. For example, the subsequent point in time maybe the point in time between the first inertial-sensor-measurement timeand the second inertial-sensor-measurement time. In someimplementations, determining the interpolated set of motion parametersmay include determining a positional change between the position of afirst depth viewpoint and a position at a subsequent point in time.

By way of non-limiting example, FIG. 7C illustrates a timeline 72 thatis similar to timeline 71 in FIG. 7B, with the addition of determination72 a and reprojection 73 a. Determination 72 a represents thedetermination of the relative motion parameters (rotational andpositional change) of depth sensor 110 between the timestamps 70 b and70 c. Reprojection 73 a represents the reprojection (of the depth image)to timestamp 70 c. By virtue of this determination, the rotation and/orposition of depth sensor 110 can be estimated at the point of time ofthe occurrence of measurement 70 c.

Reprojection component 116 may be configured to generate re-projecteddepth images representing particular depth information included inparticular depth images as if the particular depth images had beencaptured at a different point in time and/or from a different viewpoint.Reprojection component 116 may be configured to generate a re-projecteddepth image representing the depth information included in a given depthimage as if the given depth image had been captured at a particularpoint in time between a first inertial-sensor-measurement time and asecond inertial-sensor-measurement time, e.g., at the same time as aparticular image was captured by image sensor 108. In someimplementations, generation of re-projected depth images may be based onone or more interpolated motion parameters. In some implementations,generation of re-projected depth images may be based on one or morerotational changes and/or positional changes of depth sensor 110 and/orany other sensor. In some implementations, a re-projected depth imagemay use the same reference coordinate system as an image captured byimage sensor 108. In some implementations, re-projection may be based onEuclidean geometry. In some implementations, reprojection byreprojection component 116 may use a first three-dimensional point cloud(based on a first depth image captured at a first point in time),perform a three-dimensional rigid transformation to create a secondthree-dimensional point cloud (e.g., based on an estimated relativeposition and orientation of depth sensor 110 between the first point intime and a second point in time, such that the second three-dimensionalpoint cloud corresponds to the second point in time), and convert thesecond three-dimensional point cloud into a second depth image as if thesecond depth image had been captured at the second point in time (and/orfrom the second viewpoint). The second point in time may be when aparticular color image as captured. In some implementations, the rigidtransformation may be based on an estimated relative rotation andtranslation between the first and second point in time of anothersensor, such as inertial sensor 112.

Alignment component 118 may be configured to generate composite sets ofdata. In some implementations, alignment component 118 may be configuredto combine information from one or more images captured by image sensor108, one or more depth images captured by depth sensor 110, one or moremotion parameters (based on inertial signals generated and/or providedby inertial sensor 112), and/or one or more thermal images captured bythermal sensor 120. In some implementations, alignment component 118 maybe configured to generate a composite set of data by combininginformation from a color image and a re-projected depth image. In someimplementations, the composite set of data may include one or morevalues of interpolated motion parameters. For example, a composite setof data may approximate the combination of image information and depthinformation as if these had been captured at the same time and/or usingthe same viewpoint. By way of non-limiting example, FIG. 8 illustrates atimeline 80 with composite sets of data. Measurement 80 a occurs whenimage sensor 108 captures an image. By virtue of the technologiesdescribed herein, (re-projected) depth information and one or moremotion parameters may be temporally aligned with the captured imageinformation at measurement 80 a. In some implementations, the depthimage may be further re-projected by reprojection component 116 suchthat both the time and the viewpoint of the re-projected depth image of80 a coincide with the time and the viewpoint (resp.) of the colorimage. In some implementations, the absolute motion parameters of theimage sensor are determined by parameter determination component 114 andinserted into a composite set, such that all the different types ofinformation (e.g., depth, color, motion parameters) correspond to thesame point in time (the color image timestamp) and the same viewpoint(the color image viewpoint) in the reference coordinate system. Asdepicted, measurements 80 b, 80 c, 80 d, 80 e, 80 f, 80 g, 80 h, 80 i,80 j, and 80 k occur at regular intervals, indicating a regular rate ofcapture by image sensor 108. Individually captured image information maybe temporally aligned with captured depth information (in case therespective capture times naturally coincide) or re-projected depthinformation (in case a particular depth capture time does not naturallycoincide with a particular image capture time). In some implementations,individually captured image information may be temporally aligned withone or more motion parameters, and/or other parameters.

Thermal projection component 122 may be configured to generatere-projected thermal images representing thermal information included incaptured thermal image as if the captured thermal images had beencaptured at different points in time. For example, at a point in timebetween a first inertial-sensor-measurement time and a secondinertial-sensor-measurement time. In some implementations, a compositeset of data may further include information from a re-projected thermalimage. In some implementations, depth image information may be used tore-project a thermal image.

Display component 124 may be configured to present images on a display134 of an augmented reality device 132 to a user such that the user canview reality and the images simultaneously. The presented images may bebased at least in part on information included in one or more compositesets of data.

The particular and relative occurrences of different measurements inFIGS. 7A-7B-7C is exemplary and not intended to be limiting in any way.By way of non-limiting example, FIGS. 9A-9B-9C-9D illustrate exemplarytimelines with sensor operations for the use of system 100. FIG. 9Aillustrates a timeline 91 in which a measurement 90 a occurs first, whendepth sensor 110 captures a depth image, followed by measurement 90 b,when inertial sensor 112 generates a first set of signals conveyingvalues that are used to determine a first position and a firstorientation in a reference coordinate system. Subsequently, measurement90 c occurs, when image sensor 108 captures an image. Finally,measurement 90 d occurs, when inertial sensor 112 generates a second setof signals conveying values that are used to determine a second positionand a second orientation in the reference coordinate system. Through useof the technologies described herein, the information from measurements90 b and 90 d can be used (interpolated) to determine, e.g., theposition and orientation of image sensor 108 during measurement 90 c.This information may be used to re-project the captured depth image asif it has been captured at the same time as measurement 90 c. By way ofnon-limiting example, FIG. 9B illustrates a timeline 92, in whichmeasurement 90 a occurs between measurements 90 c and 90 d. Theinformation from measurements 90 b and 90 d can be used (interpolated)to determine, e.g., the position and orientation of image sensor 108during measurement 90 c. This information may be used to re-project thecaptured depth image as if it has been captured at the same time asmeasurement 90 c. By way of non-limiting example, FIG. 9C illustrates atimeline 93, in which measurement 90 a occurs after measurement 90 d.The information from measurements 90 b and 90 d can be used(interpolated) to determine, e.g., the position and orientation of imagesensor 108 during measurement 90 c. This information may be used tore-project the captured depth image as if it has been captured at thesame time as measurement 90 c.

By way of non-limiting example, FIG. 9D illustrates a timeline 94 inwhich a measurement 90 a occurs first, when inertial sensor 112generates a first set of signals, followed by measurement 90 b, whendepth sensor 110 captures a depth image. Subsequently, measurement 90 coccurs, inertial sensor 112 generates a second set of signals, followedby measurement 90 d when image sensor 108 captures an image. Finally,measurement 90 e occurs, when inertial sensor 112 generates a third setof signals. Through use of the technologies described herein, theabsolute poses (positions and orientations) from measurements 90 a and90 c can be used (indicated as an interpolation 95 a) to determine theabsolute position and orientation at the moment of measurement 90 b. Theabsolute poses from measurements 90 c and 90 e can be used (indicated asan interpolation 95 b) to determine the absolute position andorientation at the moment of measurement 90 d. The determined absoluteposes at the moments of measurements 90 b and 90 d may be used(indicated as a relative pose determination and depth re-projection 95c) to determine the relative pose from (the moments in time of)measurements 90 b and 90 d. Subsequently, this relative pose may be usedto re-project the depth image (as indicated as part of relative posedetermination and depth re-projection 95 c).

Referring to FIG. 1, in some implementations, server(s) 102, clientcomputing platform(s) 104, and/or external resources 126 may beoperatively linked via one or more electronic communication links. Forexample, such electronic communication links may be established, atleast in part, via one or more networks 13 such as, e.g., the Internetand/or other networks. It will be appreciated that this is not intendedto be limiting, and that the scope of this disclosure includesimplementations in which server(s) 102, client computing platform(s)104, and/or external resources 126 may be operatively linked via someother communication media.

A given client computing platform 104 may include one or more processorsconfigured to execute computer program components. The computer programcomponents may be configured to enable an expert or user associated withthe given client computing platform 104 to interface with system 100and/or external resources 126, and/or provide other functionalityattributed herein to client computing platform(s) 104. By way ofnon-limiting example, the given client computing platform 104 mayinclude one or more of a desktop computer, a laptop computer, a handheldcomputer, a tablet computing platform, a NetBook, a Smartphone, a gamingconsole, and/or other computing platforms.

External resources 126 may include sources of information outside ofsystem 100, external entities participating with system 100, and/orother resources. In some implementations, some or all of thefunctionality attributed herein to external resources 126 may beprovided by resources included in system 100.

Server(s) 102 may include electronic storage 128, one or more processors130, and/or other components. Server(s) 102 may include communicationlines, or ports to enable the exchange of information with a networkand/or other computing platforms. Illustration of server(s) 102 in FIG.1 is not intended to be limiting. Server(s) 102 may include a pluralityof hardware, software, and/or firmware components operating together toprovide the functionality attributed herein to server(s) 102. Forexample, server(s) 102 may be implemented by a cloud of computingplatforms operating together as server(s) 102.

Electronic storage 128 may comprise non-transitory storage media thatelectronically stores information. The electronic storage media ofelectronic storage 128 may include one or both of system storage that isprovided integrally (i.e., substantially non-removable) with server(s)102 and/or removable storage that is removably connectable to server(s)102 via, for example, a port (e.g., a USB port, a firewire port, etc.)or a drive (e.g., a disk drive, etc.). Electronic storage 128 mayinclude one or more of optically readable storage media (e.g., opticaldisks, etc.), magnetically readable storage media (e.g., magnetic tape,magnetic hard drive, floppy drive, etc.), electrical charge-basedstorage media (e.g., EEPROM, RAM, etc.), solid-state storage media(e.g., flash drive, etc.), and/or other electronically readable storagemedia. Electronic storage 128 may include one or more virtual storageresources (e.g., cloud storage, a virtual private network, and/or othervirtual storage resources). Electronic storage 128 may store softwarealgorithms, information determined by processor(s) 130, informationreceived from server(s) 102, information received from client computingplatform(s) 104, and/or other information that enables server(s) 102 tofunction as described herein.

Processor(s) 130 may be configured to provide information processingcapabilities in server(s) 102. As such, processor(s) 130 may include oneor more of a digital processor, an analog processor, a digital circuitdesigned to process information, an analog circuit designed to processinformation, a state machine, and/or other mechanisms for electronicallyprocessing information. Although processor(s) 130 is shown in FIG. 1 asa single entity, this is for illustrative purposes only. In someimplementations, processor(s) 130 may include a plurality of processingunits. These processing units may be physically located within the samedevice, or processor(s) 130 may represent processing functionality of aplurality of devices operating in coordination. Processor(s) 130 may beconfigured to execute components 114, 116, 118, 122, and/or 124, and/orother components. Processor(s) 130 may be configured to executecomponents 114, 116, 118, 122, and/or 124, and/or other components bysoftware; hardware; firmware; some combination of software, hardware,and/or firmware; and/or other mechanisms for configuring processingcapabilities on processor(s) 130. As used herein, the term “component”may refer to any component or set of components that perform thefunctionality attributed to the component. This may include one or morephysical processors during execution of processor readable instructions,the processor readable instructions, circuitry, hardware, storage media,or any other components.

It should be appreciated that although components 114, 116, 118, 122,and/or 124 are illustrated in FIG. 1 as being implemented within asingle processing unit, in implementations in which processor(s) 130includes multiple processing units, one or more of components 114, 116,118, 122, and/or 124 may be implemented remotely from the othercomponents. The description of the functionality provided by thedifferent components 114, 116, 118, 122, and/or 124 described below isfor illustrative purposes, and is not intended to be limiting, as any ofcomponents 114, 116, 118, 122, and/or 124 may provide more or lessfunctionality than is described. For example, one or more of components114, 116, 118, 122, and/or 124 may be eliminated, and some or all of itsfunctionality may be provided by other ones of components 114, 116, 118,122, and/or 124. As another example, processor(s) 130 may be configuredto execute one or more additional components that may perform some orall of the functionality attributed below to one of components 114, 116,118, 122, and/or 124.

FIG. 2 illustrates a method 200 for generating composite sets of datafrom different sensors, in accordance with one or more implementations.The operations of method 200 presented below are intended to beillustrative. In some implementations, method 200 may be accomplishedwith one or more additional operations not described, and/or without oneor more of the operations discussed. Additionally, the order in whichthe operations of method 200 are illustrated in FIG. 2 and describedbelow is not intended to be limiting.

In some implementations, method 200 may be implemented in one or moreprocessing devices (e.g., a digital processor, an analog processor, adigital circuit designed to process information, an analog circuitdesigned to process information, a state machine, and/or othermechanisms for electronically processing information). The one or moreprocessing devices may include one or more devices executing some or allof the operations of method 200 in response to instructions storedelectronically on an electronic storage medium. The one or moreprocessing devices may include one or more devices configured throughhardware, firmware, and/or software to be specifically designed forexecution of one or more of the operations of method 200.

An operation 202 may include capturing, by an image sensor, images fromviewpoints. The images may include chromatic information. The chromaticinformation of individual images may indicate one or more colorsviewable by the image sensor from individual viewpoints of the imagesensor. The images may include a first image captured at a first imagecapture time from a first image viewpoint. Operation 202 may beperformed by an image sensor that is the same as or similar to imagesensor 108, in accordance with one or more implementations.

An operation 204 may include capturing, by a depth sensor, depth imagesfrom viewpoints of the depth sensor. The depth images may include depthinformation. The depth information of individual depth images may becaptured from individual viewpoints of the depth sensor. The depthinformation of the individual depth images may indicate distances fromthe individual viewpoints to surfaces viewable by the depth sensor fromthe individual viewpoints. The depth images may include a first depthimage including first depth information. The first depth information maybe captured from a first depth viewpoint at a first depth-capture time.Operation 204 may be performed by a depth sensor that is the same as orsimilar to depth sensor 110, in accordance with one or moreimplementations.

An operation 206 may include generating, by an inertial sensor, inertialsignals that convey values that are used to determine motion parameterscharacterizing position and orientation of the inertial sensor in areference coordinate system. The inertial signals may include a firstset of inertial signals generated at a first inertial-sensor-measurementtime that convey a first set of values that is used to determine a firstset of motion parameters. The inertial signals may further include asecond set of inertial signals generated at a secondinertial-sensor-measurement time that convey a second set of values thatis used to determine a second set of motion parameters. Operation 206may be performed by an inertial sensor that is the same as or similar toinertial sensor 112, in accordance with one or more implementations.

An operation 208 may include determining the first set of values of thefirst set of one or more motion parameters based on the first set ofinertial signals, the second set of values of the second set of one ormore motion parameters based on the second set of inertial signals, andan interpolated set of values of one or more interpolated motionparameters based on the first set of values and the second set ofvalues. The interpolated set of values may correspond to a point in timebetween the first inertial-sensor-measurement time and the secondinertial-sensor-measurement time. Operation 208 may be performed by oneor more hardware processors configured by machine-readable instructionsincluding a component that is the same as or similar to parameterdetermination component 114, in accordance with one or moreimplementations.

An operation 210 may include generating a first re-projected depth imagerepresenting the first depth information included in the first depthimage as if the first depth image had been captured at the point in timebetween the first inertial-sensor-measurement time and the secondinertial-sensor-measurement time, wherein generation is based on theinterpolated set of values. Operation 210 may be performed by one ormore hardware processors configured by machine-readable instructionsincluding a component that is the same as or similar to reprojectioncomponent 116, in accordance with one or more implementations.

An operation 212 may include generating a composite set of data bycombining information from the first image, the first re-projected depthimage, and the interpolated set of values of the one or moreinterpolated motion parameters. Operation 212 may be performed by one ormore hardware processors configured by machine-readable instructionsincluding a component that is the same as or similar to alignmentcomponent 118, in accordance with one or more implementations.

Although the present technology has been described in detail for thepurpose of illustration based on what is currently considered to be themost practical and preferred implementations, it is to be understoodthat such detail is solely for that purpose and that the technology isnot limited to the disclosed implementations, but, on the contrary, isintended to cover modifications and equivalent arrangements that arewithin the spirit and scope of the appended claims. For example, it isto be understood that the present technology contemplates that, to theextent possible, one or more features of any implementation can becombined with one or more features of any other implementation.

What is claimed is:
 1. A system configured for generating composite setsof data based on sensor data from different sensors, the systemcomprising: one or more hardware processors configured bymachine-readable instructions to: capture, by an image sensor, imagesfrom viewpoints, the images including chromatic information, thechromatic information of individual images indicating one or more colorsviewable by the image sensor from individual viewpoints of the imagesensor, wherein the images include a first image captured at a firstimage capture time from a first image viewpoint; capture, by a depthsensor, depth images from viewpoints of the depth sensor, the depthimages including depth information, wherein the depth information ofindividual depth images is captured from individual viewpoints of thedepth sensor, wherein the depth information of the individual depthimages indicates distances from the individual viewpoints of the depthsensor to surfaces viewable by the depth sensor from the individualviewpoints, wherein the depth images includes a first depth imageincluding first depth information, wherein the first depth informationis captured from a first depth viewpoint at a first depth-capture time,wherein the first depth information indicates a first set of distancesfrom the first depth viewpoint to the surfaces; generate, by an inertialsensor, inertial signals that convey values that are used to determinemotion parameters characterizing position and orientation of theinertial sensor in a reference coordinate system, wherein the inertialsignals include a first set of inertial signals and a second set ofinertial signals, wherein the first set of inertial signals is generatedat a first inertial-sensor-measurement time and conveys a first set ofvalues that is used to determine a first set of motion parameters,wherein the second set of inertial signals is generated at a secondinertial-sensor-measurement time and conveys a second set of values thatis used to determine a second set of motion parameters; determine thefirst set of values of the first set of one or more motion parametersbased on the first set of inertial signals, the second set of values ofthe second set of one or more motion parameters based on the second setof inertial signals, and an interpolated set of values of one or moreinterpolated motion parameters based on the first set of values and thesecond set of values, wherein the interpolated set of values correspondsto a point in time between the first inertial-sensor-measurement timeand the second inertial-sensor-measurement time; generate a firstre-projected depth image representing the first depth informationincluded in the first depth image as if the first depth image had beencaptured at the point in time between the firstinertial-sensor-measurement time and the secondinertial-sensor-measurement time, wherein generation is based on theinterpolated set of values; and generate a composite set of data bycombining information from the first image, the first re-projected depthimage, and the interpolated set of values.
 2. The system of claim 1,wherein the first inertial-sensor-measurement time occurs before thefirst image capture time, and wherein the secondinertial-sensor-measurement time occurs after the first image capturetime.
 3. The system of claim 1, wherein the first set of values of thefirst set of one or more motion parameters includes a first position anda first orientation in the reference coordinate system, wherein thesecond set of values of the second set of one or more motion parametersincludes a second position and a second orientation in the referencecoordinate system, and wherein the interpolated set of values include aninterpolated position and an interpolated orientation in the referencecoordinate system.
 4. The system of claim 3, wherein the point in timeof the interpolated set of values coincides with the first image capturetime.
 5. The system of claim 1, wherein determining the interpolated setof values includes determining a rotational change between anorientation of the depth sensor at the first depth viewpoint and aninterpolated orientation of the depth sensor at the point in timebetween the first inertial-sensor-measurement time and the secondinertial-sensor-measurement time.
 6. The system of claim 5, whereindetermining the interpolated set of values includes determining apositional change between a first depth position and an interpolatedposition at the point in time between the firstinertial-sensor-measurement time and the secondinertial-sensor-measurement time.
 7. The system of claim 6, whereingenerating the first re-projected depth image is based on the rotationalchange and the positional change.
 8. The system of claim 1, wherein themotion parameters include one or more of angular velocity and/oracceleration.
 9. The system of claim 1, wherein the one or more hardwareprocessors are further configured by machine-readable instructions to:capture thermal images including thermal information, wherein thethermal images include a first thermal image captured from a particularviewpoint and a particular capture time; generate a first re-projectedthermal image representing thermal information included in the firstthermal image as if the first thermal image had been captured at thepoint in time between the first inertial-sensor-measurement time and thesecond inertial-sensor-measurement time; wherein the composite set ofdata further includes information from the first re-projected thermalimage.
 10. The system of claim 1, wherein the one or more hardwareprocessors are further configured by machine-readable instructions to:present images on a display of an augmented reality device to a usersuch that the user can view reality and the images simultaneously,wherein the presented images are based at least in part on informationincluded in the composite set of data.
 11. A method for generatingcomposite sets of data based on sensor data from different sensors, themethod comprising: capturing, by an image sensor, images fromviewpoints, the images including chromatic information, the chromaticinformation of individual images indicating one or more colors viewableby the image sensor from individual viewpoints of the image sensor,wherein the images include a first image captured at a first imagecapture time from a first image viewpoint; capturing, by a depth sensor,depth images from viewpoints of the depth sensor, the depth imagesincluding depth information, wherein the depth information of individualdepth images is captured from individual viewpoints of the depth sensor,wherein the depth information of the individual depth images indicatesdistances from the individual viewpoints of the depth sensor to surfacesviewable by the depth sensor from the individual viewpoints, wherein thedepth images includes a first depth image including first depthinformation, wherein the first depth information is captured from afirst depth viewpoint at a first depth-capture time, wherein the firstdepth information indicates a first set of distances from the firstdepth viewpoint to the surfaces; generating, by an inertial sensor,inertial signals that convey values that are used to determine motionparameters characterizing position and orientation of the inertialsensor in a reference coordinate system, wherein the inertial signalsinclude a first set of inertial signals and a second set of inertialsignals, wherein the first set of inertial signals is generated at afirst inertial-sensor-measurement time and conveys a first set of valuesthat is used to determine a first set of motion parameters, and whereinthe second set of inertial signals is generated at a secondinertial-sensor-measurement time and conveys a second set of values thatis used to determine a second set of motion parameters; determining thefirst set of values of the first set of one or more motion parametersbased on the first set of inertial signals, the second set of values ofthe second set of one or more motion parameters based on the second setof inertial signals, and an interpolated set of values of one or moreinterpolated motion parameters based on the first set of values and thesecond set of values, wherein the interpolated set of values correspondsto a point in time between the first inertial-sensor-measurement timeand the second inertial-sensor-measurement time; generating a firstre-projected depth image representing the first depth informationincluded in the first depth image as if the first depth image had beencaptured at the point in time between the firstinertial-sensor-measurement time and the secondinertial-sensor-measurement time, wherein generation is based on theinterpolated set of values; and generating a composite set of data bycombining information from the first image, the first re-projected depthimage, and the interpolated set of values.
 12. The method of claim 11,wherein the first inertial-sensor-measurement time occurs before thefirst image capture time, and wherein the secondinertial-sensor-measurement time occurs after the first image capturetime.
 13. The method of claim 11, wherein the first set of values of thefirst set of one or more motion parameters includes a first position anda first orientation in the reference coordinate system, wherein thesecond set of values of the second set of one or more motion parametersincludes a second position and a second orientation in the referencecoordinate system, and wherein the interpolated set of values include aninterpolated position and an interpolated orientation in the referencecoordinate system.
 14. The method of claim 13, wherein the point in timeof the interpolated set of values coincides with the first image capturetime.
 15. The method of claim 11, wherein determining the interpolatedset of values includes determining a rotational change between anorientation of the depth sensor at the first depth viewpoint and aninterpolated orientation of the depth sensor at the point in timebetween the first inertial-sensor-measurement time and the secondinertial-sensor-measurement time.
 16. The method of claim 15, whereindetermining the interpolated set of values includes determining apositional change between a first depth position and an interpolatedposition at the point in time between the firstinertial-sensor-measurement time and the secondinertial-sensor-measurement time.
 17. The method of claim 16, whereingenerating the first re-projected depth image is based on the rotationalchange and the positional change.
 18. The method of claim 11, whereinthe motion parameters include one or more of angular velocity and/oracceleration.
 19. The method of claim 11, further comprising: capturingthermal images including thermal information, wherein the thermal imagesinclude a first thermal image captured from a particular viewpoint and aparticular capture time; generating a first re-projected thermal imagerepresenting thermal information included in the first thermal image asif the first thermal image had been captured at the point in timebetween the first inertial-sensor-measurement time and the secondinertial-sensor-measurement time; wherein the composite set of datafurther includes information from the first re-projected thermal image.20. The method of claim 11, further comprising: presenting images on adisplay of an augmented reality device to a user such that the user canview reality and the images simultaneously, wherein the presented imagesare based at least in part on information included in the composite setof data.